As the world becomes more interconnected, digitized, and globalized, service providers are facing an enormous surge in traffic that will continue to grow exponentially. In the face of exploding data volumes and shrinking decision time windows, service providers have to make key business decisions in real-time to remain competitive. Not only in business but also in many science and engineering fields, fast insight into complex processes is needed. In telemetry, data is collected from distributed sources and transmitted to central aggregation points for monitoring, management and data analytics purposes. Telemetry applications range from meteorology, oil and gas industry, traffic surveillance, space industry, to motor racing and agriculture. In these applications, real-time analytics is helpful since it allows to immediately and continuously monitor effects that certain changes entail and to react in time when needed. Gathering real-time insight from data generated across distributed systems provides significant benefits for businesses by being able to react faster to changes and customer demands. In addition, devices (typically the edges of large distributed systems) are getting smarter and smarter and, correspondingly, able to produce more complex and larger volumes of data streams. Many connected devices scenarios benefit from gathering real-time insights: logistic, manufacturing, power utilities, telematics, data center monitoring, etc. Unfortunately, current tools and technologies designed to aid decision-making can no longer meet their needs. These tools require data to be recorded on a storage device, followed by offline analytic processing, to detect actionable insights. This is a time-consuming process and the real-time aspect often gets lost.
The problem arises of efficiently gathering data streams from devices to enable genuine real-time analytics of all these data streams in a cloud environment.
Previous solutions have relied on (persistently) storing data first and only then performing the analytics. For some scenarios this introduces unacceptable latency.
The embodiments described below are not limited to implementations which solve any or all of the problems mentioned above.